fix: add exact resume-safe WWPGD projection dose telemetry - #131

Status

Ready for review. This PR supersedes the scientific-runner intent of #129 without carrying forward its off-by-one, first-application, or long-run performance errors. All six repository CI jobs pass on the exact head.

Scope

The change applies to wwpgd_projection.csv artifacts written by the main run_scientific_single durable writer. It covers the scientific Level 0–2 WW-PGD, delayed-onset, adaptive, and norm-matched-sham arms.

It deliberately does not silently redefine the separate invalid-for-science run_single smoke CSV or the isolated level_0_wwpgd package’s independent schema; those paths have different lifecycle conventions.

What changed

Every main scientific-runner row written to wwpgd_projection.csv now includes:

• dose_definition = applied_projection_delta_frobenius_over_preprojection_weight_frobenius
• dose_value, equal to the existing realized relative_frobenius_change_applied
• dose_relative_frobenius, a compatibility alias of dose_value
• dose_applied, true only when the live matrix actually changed and the realized dose is positive
• layer_application_index, a one-based count of actual applications for that layer
• is_first_applied_projection = (layer_application_index == 1)
• projection_event_index_base = 0
• projection_event_number = projection_event + 1
• is_first_scheduled_projection_event = (projection_event == 0)

The realized dose is

||W_after - W_before||_F / max(||W_before||_F, 1e-12)

Why this differs from #129

PR #129 marked event_index == 1 as the first projection event, but the main scientific training extension uses zero-based event indexes. More importantly, a scheduled event is not necessarily an actual application: warm-up, delayed onset, adaptive gates, zero hardness, unchanged candidates, and trust-region behavior can delay or suppress a layer update.

This PR therefore separates two facts:

• the first scheduled scientific event is event zero;
• the first actual application is the first positive realized dose that changes a particular layer.

Actual applications are counted independently per layer from the durable projection history. Checkpoint recovery reconciles the CSV to its committed prefix before new writes, making is_first_applied_projection correct after resume or process restart without adding mutable counters to the optimizer, controller, model, or checkpoint state.

Efficient durable index

A small hidden sidecar caches only the current per-layer application counts plus the projection CSV byte size and nanosecond modification time. Normal interval-one experiments therefore do not rescan the complete CSV after every optimizer step.

The CSV remains authoritative. If the sidecar is absent, malformed, stale, or no longer matches a reconciled/truncated CSV, the cache is discarded and rebuilt from the durable CSV before the next append.

Implementation boundary

The enrichment occurs in the shared durable CSV writer only when the target file is exactly wwpgd_projection.csv. Other scientific CSV artifacts are unchanged, and caller row dictionaries are copied rather than mutated.

The writer fails loudly for malformed projection telemetry, including:

• missing, fractional, or negative event indexes;
• missing layer identities;
• missing, negative, or non-finite applied relative Frobenius values;
• invalid boolean values;
• the ambiguous legacy is_first_projection_event field;
• attempts to append to an older projection CSV schema.

Compatibility

This extends the main scientific projection CSV header. Existing incomplete scientific runs with the old header are rejected before append and should be restarted under the new schema. No model weights, optimizer state, projection mathematics, target alpha, hardness, trust-region control, or defaults change.

Validation

Focused tests cover:

• exact dose identity and aliases;
• correct zero-based first-scheduled-event labeling;
• a first scheduled event with zero dose followed by a later first actual application;
• one-based per-layer application counts across sequential durable appends;
• independent first-application state for different layers;
• fast-path sidecar use without full-history rescans;
• malformed/stale sidecar rebuilding;
• checkpoint reconciliation of an uncommitted CSV suffix before the next application count;
• no mutation of caller rows;
• no changes to non-projection CSVs;
• fail-loud behavior for malformed telemetry and old schemas;
• checkpoint/resume and cadence test doubles using the production projection-row contract.

All six repository CI jobs pass on the exact head:

• complete portable test suite;
• quality/compilation/Ruff;
• Level 0–2 local runtime acceptance;
• macOS Python 3.10 / NumPy 1.x release acceptance;
• resolved pilot manifests;
• schema-v3 Papermill notebooks.

No scientific efficacy claim is made.